Developing and Validating the Artificial Intelligence Literacy Concept Inventory: an Instrument to Assess Artificial Intelligence Literacy among Middle School Students
Open accessAbstract
The rapid expansion of Artificial Intelligence (AI) in our society makes it urgent and necessary to develop young students’ AI literacy so that they can become informed citizens and critical consumers of AI technology. Over the past decade many efforts have focused on developing curricular materials that make AI concepts accessible and engaging to young learners; and yet, limited research investigated how to assess learners’ AI literacy, which is critically important to inform the teaching and learning of AI. This paper addresses this issue by reporting the development and validation findings of the AI Literacy Concept Inventory Assessment (AI-CI), a set of multiple-choice questions designed to assess understanding of AI literacy concepts among middle school students. The AI-CI consists of 20 multiple choice questions examining student understanding of four topics: AI general concepts, logic systems, machine learning general concepts, and supervised learning. The content validity of AI-CI was established through multiple rounds of expert panel reviews with AI educators and experts, observations of student learning of AI, and cognitive validation interviews. The validity of the AI-CI was established with a sample of 981 students and the pre-posttest reliability was established with a sample of 108 middle school students who learned AI through experiencing the Developing AI literacy (DAILy) curriculum. The findings show that the AI-CI is a valid and reliable tool to assess AI literacy at the middle school level.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI-CI, AILQ, and Kim & Lee's instrument all provided evidence for structural validity.
“When aiming for an assessment of secondary education students, AI-CI33, AILQ35 and Kim & Lee’s instrument41 all provided evidence for content validity, structural validity, and internal consistency, although AI-CI33 and AILQ35 had higher level of evidence for content validity and provided evidence for responsiveness.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
AI-CI, AILQ, and Kim & Lee's instrument all provided evidence for content validity.
“When aiming for an assessment of secondary education students, AI-CI33, AILQ35 and Kim & Lee’s instrument41 all provided evidence for content validity, structural validity, and internal consistency, although AI-CI33 and AILQ35 had higher level of evidence for content validity and provided evidence for responsiveness.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
AI-CI, AILQ, and Kim & Lee's instrument all provided evidence for internal consistency.
“When aiming for an assessment of secondary education students, AI-CI33, AILQ35 and Kim & Lee’s instrument41 all provided evidence for content validity, structural validity, and internal consistency, although AI-CI33 and AILQ35 had higher level of evidence for content validity and provided evidence for responsiveness.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
AI-CI and AILQ had higher level of evidence for content validity than Kim & Lee's instrument.
“When aiming for an assessment of secondary education students, AI-CI33, AILQ35 and Kim & Lee’s instrument41 all provided evidence for content validity, structural validity, and internal consistency, although AI-CI33 and AILQ35 had higher level of evidence for content validity and provided evidence for responsiveness.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
AI-CI and AILQ provided evidence for responsiveness.
“When aiming for an assessment of secondary education students, AI-CI33, AILQ35 and Kim & Lee’s instrument41 all provided evidence for content validity, structural validity, and internal consistency, although AI-CI33 and AILQ35 had higher level of evidence for content validity and provided evidence for responsiveness.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
AI-CI and AILQ are currently available only in English.
“The decision between the two instruments might, to some degree, be guided by the languages they are available in, with AI-CI33 and AILQ35 currently available only in English, and Kim & Lee’s instrument41 only in Korean.”
From A systematic review of AI literacy scales (2024) · cites this paper as [33]
Cites
Messick, S. (1995) · American Psychologist · doi:10.1037/0003-066x.50.9.741 · Cited by 4,198 · see citing works
Scale Development: Theory and Applications.
Xie, Y., & DeVellis, R. F. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075704 · Cited by 3,691 · see citing works
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 2× · see citing works
“For instance, Long and Magerko (2020) through a synthesis of the published work provided a concrete definition of AI literacy that included a comprehensive set of competencies “that enables individuals to critically evaluate AI technologies, effectively communicate and collaborate with AI, and…”
Hestenes, D., Wells, M., & Swackhamer, G. (1992) · The Physics Teacher · doi:10.1119/1.2343497 · Cited by 3,235 · see citing works
Sampling in Interview-Based Qualitative Research: A Theoretical and Practical Guide
Robinson, O. C. (2014) · Qualitative Research in Psychology · doi:10.1080/14780887.2013.801543 · Cited by 3,232 · see citing works
Peer Instruction: Ten years of experience and results
Crouch, C. H., & Mazur, E. (2001) · American Journal of Physics · doi:10.1119/1.1374249 · Cited by 2,682 · see citing works
ltm : An R Package for Latent Variable Modeling and Item Response Theory Analyses
Rizopoulos, D. (2006) · Journal of Statistical Software · doi:10.18637/jss.v017.i05 · Cited by 1,246 · see citing works
Envisioning AI for K-12: What Should Every Child Know about AI?
Touretzky, D., Gardner-McCune, C., Martin, F., & Seehorn, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019795 · Cited by 918 · see citing works
ASSESSMENT – SUMMATIVE AND FORMATIVE – SOME THEORETICAL REFLECTIONS
Taras, M. (2005) · British Journal of Educational Studies · doi:10.1111/j.1467-8527.2005.00307.x · Cited by 727 · see citing works
Multiple surveys of students and survey fatigue
Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004) · New Directions for Institutional Research · doi:10.1002/ir.101 · Cited by 541 · see citing works
Mazur, E. (2009) · Science · doi:10.1126/science.1168927 · Cited by 473 · see citing works
Long-Term Trends in the Public Perception of Artificial Intelligence
Fast, E., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10635 · Cited by 467 · cited 1× · see citing works
“This suggests that some middle school students have recognized that AI includes more technologies or tools besides robots, which is different from the public’s perception of AI as robot as reported in the literature (Fast & Horvitz, 2017; Ottenbreit-Leftwich et al., 2021; Zimmerman, 2018).”
Development and Validation of Instruments to Measure Learning of Expert‐Like Thinking
Adams, W. K., & Wieman, C. (2010) · International Journal of Science Education · doi:10.1080/09500693.2010.512369 · Cited by 460 · see citing works
Artificial intelligence and computer science in education: From kindergarten to university
Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016) · 2016 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2016.7757570 · Cited by 437 · see citing works
Advances in Motivation and Achievement
(2010) · Advances in motivation and achievement · doi:10.1108/s0749-7423(2010)000016a013 · Cited by 435 · see citing works
Zhang, H., Lee, I., Ali, S., DiPaola, D., Cheng, Y., & Breazeal, C. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00293-3 · Cited by 396 · see citing works
Self-citationCognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., BLAZEVSKI, J., Bonney, C. R., de Groot, E., Gilbert, M. C., Musu, L. E., Kempler, T. M., et al. (2007) · Educational Psychologist · doi:10.1080/00461520701416231 · Cited by 385 · see citing works
"Hey Google is it OK if I eat you?": Initial Explorations in Child-Agent Interaction
Druga, S., Williams, R., Breazeal, C., & Resnick, M. (2017) · Proceedings of the 2017 Conference on Interaction Design and Children · doi:10.1145/3078072.3084330 · Cited by 330 · see citing works
Diagnostic Assessment With Ordered Multiple-Choice Items
Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. R. (2006) · Educational Assessment · doi:10.1207/s15326977ea1101_2 · Cited by 285 · see citing works
Çelik, İ. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102026 · Cited by 276 · see citing works
A is for Artificial Intelligence
Williams, R., Park, H. W., & Breazeal, C. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300677 · Cited by 274 · see citing works
EURASIA Journal of Mathematics, Science and Technology Education
(2019) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste · Cited by 253 · see citing works
Garvin-Doxas, K., & Klymkowsky, M. W. (2008) · CBE—Life Sciences Education · doi:10.1187/cbe.07-08-0063 · Cited by 216 · see citing works
AI + Ethics Curricula for Middle School Youth: Lessons Learned from Three Project-Based Curricula
Williams, R., Ali, S., Devasia, N., DiPaola, D., Hong, J., Kaputsos, S. P., Jordan, B., & Breazeal, C. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00298-y · Cited by 206 · see citing works
Eguchi, A., Okada, H., & Muto, Y. (2021) · Künstliche Intell. · doi:10.1007/s13218-021-00737-3 · Cited by 202 · see citing works
An artificial intelligence educational strategy for the digital transformation
Cantú-Ortiz, F. J., Galeano Sánchez, N., Garrido, L., Terashima-Marin, H., & Brena, R. F. (2020) · International Journal on Interactive Design and Manufacturing (IJIDeM) · doi:10.1007/s12008-020-00702-8 · Cited by 192 · see citing works
Drasgow, F., & Lissak, R. I. (1983) · Journal of Applied Psychology · doi:10.1037/0021-9010.68.3.363 · Cited by 177 · see citing works
Computer science concept inventories: past and future
Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B., & Clancy, M. J. (2014) · Computer Science Education · doi:10.1080/08993408.2014.970779 · Cited by 129 · see citing works
Using the Force Concept Inventory to monitor student learning and to plan teaching
Savinainen, A., & Scott, P. (2002) · Physics Education · doi:10.1088/0031-9120/37/1/307 · Cited by 125 · see citing works
Setting the Scope of Concept Inventories for Introductory Computing Subjects
Goldman, K., Gross, P., Heeren, C., Herman, G. L., Kaczmarczyk, L. C., Loui, M. C., & Zilles, C. B. (2010) · ACM Transactions on Computing Education · doi:10.1145/1789934.1789935 · Cited by 122 · see citing works
Developing a Computer Science Concept Inventory for Introductory Programming
Caceffo, R., Wolfman, S., Booth, K. S., & Azevedo, R. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844559 · Cited by 119 · see citing works
Sadler, P. M., Coyle, H. P., Miller, J. L., et al. (2009) · Astronomy Education Review · doi:10.3847/aer2009024 · Cited by 106 · see citing works
An Analytic Framework for Evaluating the Validity of Concept Inventory Claims
Jorion, N., Gane, B. D., James, K., Schroeder, L., DiBello, L. V., & Pellegrino, J. W. (2015) · Journal of Engineering Education · doi:10.1002/jee.20104 · Cited by 96 · see citing works
Testing the test: Item response curves and test quality
Morris, G. A., Branum‐Martin, L., Harshman, N. L., Baker, S. D., Mazur, E., Dutta, S. N., Mzoughi, T., & McCauley, V. (2006) · American Journal of Physics · doi:10.1119/1.2174053 · Cited by 88 · see citing works
Exploring Generative Models with Middle School Students
Ali, S., DiPaola, D., Lee, I., Hong, J., & Breazeal, C. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445226 · Cited by 82 · see citing works
Self-citation(2018) · Cited by 73 · see citing works
Development, testing, and application of a chemistry concept inventory
Krause, S., Birk, J. P., Bauer, R. A., Jenkins, B., & Pavelich, M. J. (2005) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2004.1408473 · Cited by 64 · see citing works
Bock, R. D., & Gibbons, R. D. (2021) · Wiley eBooks · doi:10.1002/9781119716723 · Cited by 61 · see citing works
Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2006) · Educational and Psychological Measurement · doi:10.1177/0013164405282467 · Cited by 49 · see citing works
Identity development and perceived parental behavior as sources of adolescent egocentrism
O’Connor, B. P. (1995) · Journal of Youth and Adolescence · doi:10.1007/bf01537150 · Cited by 46 · see citing works
Distractor Similarity and Item-Stem Structure: Effects on Item Difficulty
Ascalon, M. E., Meyers, L. S., Davis, B. W., et al. (2007) · Applied Measurement in Education · doi:10.1080/08957340701301272 · Cited by 41 · see citing works
Concept inventories in computer science for the topic discrete mathematics
Almstrum, V. L., Henderson, P. B., Harvey, V., et al. (2006) · doi:10.1145/1189215.1189182 · Cited by 34 · see citing works
Wiebe, E., Rachmatullah, A., Akram, B., Boulden, D. C., Mott, B., Boyer, K. E., & Lester, J. C. (2020) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/116600 · Cited by 28 · see citing works
(1999) · Assessment · doi:10.1177/107319119900600111 · Cited by 28 · see citing works
Students’ Conceptions of Artificial Intelligence
Kreinsen, M., & Schulz, S. (2021) · doi:10.1145/3481312.3481328 · Cited by 26 · see citing works
How do Elementary Students Conceptualize Artificial Intelligence?
Ottenbreit-Leftwich, A. T., Glazewski, K. D., Jeon, M., Hmelo‐Silver, C. E., Mott, B., Lee, S., & Lester, J. C. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3439642 · Cited by 25 · see citing works
Understanding performance in test taking: The role of question difficulty order
Anaya, L., Iriberri, N., Rey‐Biel, P., & Zamarro, G. (2022) · Economics of Education Review · doi:10.1016/j.econedurev.2022.102293 · Cited by 21 · see citing works
The Practical Details of Building a CS Concept Inventory
Taylor, C., Clancy, M. J., Webb, K. C., Zingaro, D., Lee, C. B., & Porter, L. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366903 · Cited by 19 · see citing works
Development and validation of the moon phases concept inventory for middle school
Chastenay, P., & Riopel, M. (2020) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.16.020107 · Cited by 12 · see citing works
Artificial Intelligence in Pre-University Education: What and How to Teach
Guerreiro-Santalla, S., Bellas, F., & Duro, R. J. (2020) · doi:10.3390/proceedings2020054048 · Cited by 9 · see citing works
Sriram, K. (2023) · British Journal of Multidisciplinary and Advanced Studies · doi:10.37745/bjmas.2022.0363 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A systematic review of AI literacy scales
Lintner, T. (2024) · npj Science of Learning · doi:10.1038/s41539-024-00264-4 · Cited by 292 · see citing works
Wijaya, T. T., Yu, Q., Cao, Y., He, Y., & Leung, F. K. S. (2024) · Behavioral Sciences · doi:10.3390/bs14111008 · Cited by 103 · see citing works
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
Learning About AI: A Systematic Review of Reviews on AI Literacy
Zhang, S., Ganapathy Prasad, P., & Schroeder, N. L. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251342081 · Cited by 62 · see citing works
- Authors
- Zhang, H., Perry, A., & Lee, I.
- Affiliations
- Boston College (US)
- University of North Dakota (US)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Zhang, Helen
- Venue
- International Journal of Artificial Intelligence in Education
- Publisher
- Elsevier BV
- Published
- 2025-03
- Volume
- 35
- Issue
- 1
- Pages
- 398-438
- ISSN
- 1560-4292 (ISSN-L)
- OpenAlex
- W4396655318
- Cited by
- 75
- References
- 51
- Citations by year
- 2026: 41 · 2025: 32 · 2024: 2
- FWCI
- 28.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Artificial Intelligence in Education (publishedVersion, cc-by-nc-nd)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Funders
- National Science Foundation (2022502, 2048746)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Digital literacy in education
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- AI literacy, scale development, junior high school students, artificial intelligence in education, content validity, multiple-choice assessment, supervised learning, machine learning concepts, logical systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
