Evaluating the Effectiveness of Educational Augmented Reality Authoring Tool: Insights From Instructors, Experts, and Students
Abstract
This article assessed the effectiveness of an augmented reality (AR) authoring tool for aviation weather training developed to support flight instructors without requiring technical expertise. While AR-based training showed promise in aviation weather education, there are limited tools available for nontechnologist instructors. To address this challenge, an instructor-centered AR authoring tool was developed and evaluated using a multiphase framework. The AR authoring approach was validated on three dimensions: instructors’ authoring experience; instructional quality of produced AR lessons; and students’ learning outcomes and engagement. Flight instructors successfully developed interactive AR lessons on aviation weather topics of their choice, and reported high usability and manageable workload for authoring. Instructors reported increased confidence in creating AR content, while their concerns regarding technical complexity, mental effort, and time demands decreased. Subject matter experts rated the instructional quality of instructor-authored AR lessons highly. Using the produced AR lessons, students showed significant gains in knowledge and motivation. These findings demonstrate the potential of instructor-centered authoring approach to enable the creation of high-quality, interactive educational content without requiring technical expertise, and to provide a replicable, evidence-based framework for assessing educational technologies and supporting broader integration of AR in education.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
Azuma, R. T. (1997) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres.1997.6.4.355 · Cited by 10,121 · see citing works
Augmented reality: a class of displays on the reality-virtuality continuum
Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.197321 · Cited by 2,374 · see citing works
Dunleavy, M., Dede, C., & Mitchell, R. (2008) · Journal of Science Education and Technology · doi:10.1007/s10956-008-9119-1 · Cited by 1,526 · see citing works
Augmented reality for STEM learning: A systematic review
Ibáñez, M.-B., & Delgado-Kloos, C. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.05.002 · Cited by 1,036 · see citing works
Krantz, S. G. (1999) · doi:10.5948/upo9781614440260.011 · Cited by 714 · see citing works
Huang, T.-C., Chen, C.-C., & Chou, Y.-W. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.02.008 · Cited by 453 · see citing works
Teacher Digital Literacy: The Indisputable Challenge after COVID-19
Sánchez-Cruzado, C., Campión, R. S., & Compaña, M. T. S. (2021) · Sustainability · doi:10.3390/su13041858 · Cited by 343 · see citing works
Sauro, J., & Lewis, J. R. (2016) · Quantifying the User Experience · doi:10.1016/b978-0-12-802308-2.00002-3 · Cited by 312 · see citing works
Alalwan, N., Cheng, L. K., Al‐Samarraie, H., Yousef, R., Alzahrani, A. I., & Sarsam, S. M. (2020) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2020.100876 · Cited by 279 · see citing works
Ashtari, N., Bunt, A., McGrenere, J., Nebeling, M., & Chilana, P. K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376722 · Cited by 278 · see citing works
MacIntyre, B., Gandy, M., Dow, S. P., & Bolter, J. D. (2004) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1029632.1029669 · Cited by 264 · see citing works
Loorbach, N., Peters, O., Karreman, J., & Steehouder, M. F. (2014) · British Journal of Educational Technology · doi:10.1111/bjet.12138 · Cited by 243 · see citing works
Augmented Reality Applications in Education: Teachers Point of View
Tzima, S., Styliaras, G., & Bassounas, A. (2019) · Education Sciences · doi:10.3390/educsci9020099 · Cited by 216 · see citing works
The Trouble with Augmented Reality/Virtual Reality Authoring Tools
Nebeling, M., & Speicher, M. (2018) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct.2018.00098 · Cited by 169 · see citing works
Krauß, V., Boden, A., Oppermann, L., & Reiners, R. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445335 · Cited by 133 · see citing works
Meta-AR-App: An Authoring Platform for Collaborative Augmented Reality in STEM Classrooms
Villanueva, A., Zhu, Z., Liu, Z., Peppler, K., Redick, T. S., & Karthik Ramani, A. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376146 · Cited by 81 · see citing works
Augmented reality via expert demonstration authoring (AREDA)
Bhattacharya, B., & Winer, E. H. (2018) · Computers in Industry · doi:10.1016/j.compind.2018.04.021 · Cited by 72 · see citing works
Self-citationA Review on Augmented Reality Authoring Toolkits for Education
Dengel, A., Iqbal, M. Z., Grafe, S., & Mangina, E. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.798032 · Cited by 70 · see citing works
AMIRE - authoring mixed reality
Grimm, P., Haller, M., Paelke, V., Reinhold, S., Reimann, C., & Zauner, R. J. (2003) · IEEE International Workshop on Augmented Reality Toolkit (ART) · doi:10.1109/art.2002.1107008 · Cited by 70 · see citing works
An augmented reality-based authoring tool for E-learning applications
Jee, H.-K., Lim, S., Youn, J., & Lee, J.-S. (2011) · Multimedia Tools and Applications · doi:10.1007/s11042-011-0880-4 · Cited by 59 · see citing works
A pragmatic approach to augmented reality authoring
Haringer, M., & Regenbrecht, H. T. (2003) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2002.1115093 · Cited by 47 · see citing works
Ley, T., Maier, R. K., Thalmann, S., Waizenegger, L., Pata, K., & Ruiz‐Calleja, A. (2019) · Vocations and Learning · doi:10.1007/s12186-019-09231-2 · Cited by 43 · see citing works
Czok, V., Krug, M., Müller, S., Huwer, J., Kruse, S., Müller, W., & Weitzel, H. (2023) · Education Sciences · doi:10.3390/educsci13090926 · Cited by 39 · see citing works
Ramirez, H., Mendívil, E. G., Flores, P. R., & Contero, M. (2013) · Procedia Computer Science · doi:10.1016/j.procs.2013.11.023 · Cited by 36 · see citing works
A Comparison of Desktop and Augmented Reality Scenario Based Training Authoring Tools
Gonzalez, A. V., Koh, S., Kapalo, K. A., Sottilare, R. A., Garrity, P., Billinghurst, M., & LaViola, J. J. (2019) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2019.00032 · Cited by 21 · see citing works
Yang, K. B., Zhou, X., & Radu, I. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13779 · Cited by 20 · see citing works
Educators’ opinions about VR/AR/XR: An exploratory study
Schwaiger, M., Krajnčan, M., Vukovič, M., Jenko, M., & Doz, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12808-7 · Cited by 18 · see citing works
Mobile Augmented Reality Authoring Tool
Yang, Y., Shim, J., Chae, S., & Han, T. (2016) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2016.42 · Cited by 15 · see citing works
VFR-into-IMC Accident Trends: Perceptions of Deficiencies in Training
Major, W. L., Carney, T., Keller, J. C., Xie, A., Price, M., Duncan, J., Brown, L. J., Whitehurst, G., Rantz, W. G., Nicolai, D., et al. (2017) · Journal of Aviation Technology and Engineering · doi:10.7771/2159-6670.1153 · Cited by 15 · see citing works
Self-citationEmpowering Instructors: Augmented Reality Authoring Toolkit for Aviation Weather Education
Kim, J. W., Miller, J. R., Wang, K., Dorneich, M. C., Winer, E. H., & Brown, L. J. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3486630 · Cited by 14 · see citing works
Self-citationHow to Evaluate Augmented Reality Embedded in Lesson Planning in Teacher Education
Henne, A., Syskowski, S., Krug, M., Möhrke, P., Thoms, L., & Huwer, J. (2024) · Education Sciences · doi:10.3390/educsci14030264 · Cited by 11 · see citing works
Kim, J. W., Wang, K., Miller, J. R., Dorneich, M. C., Winer, E. H., Brown, L. J., & Caldwell, B. S. (2024) · Ergonomics · doi:10.1080/00140139.2024.2329696 · Cited by 11 · see citing works
Self-citationAugmented Reality Enhanced Thunderstorm Learning Experiences for General Aviation
Meister, P. P., Wang, K., Dorneich, M. C., Winer, E. H., Brown, L. J., & Whitehurst, G. (2022) · Journal of Air Transportation · doi:10.2514/1.d0308 · Cited by 8 · see citing works
Self-citationA Unified Framework for Remote Collaboration Using Interactive AR Authoring and Hands Tracking
Yu, J., Jeon, J.-U., Park, G., Kim, H.-I., & Woo, W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-39862-4_13 · Cited by 8 · see citing works
Hönemann, K., Konopka, B., & Wiesche, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-32808-4_6 · Cited by 6 · see citing works
Evaluation of Augmented Reality Interactive Print for General Aviation Weather Training
Meister, P. P., Wang, K., Dorneich, M. C., Winer, E. H., Brown, L. J., & Whitehurst, G. (2023) · Journal of Air Transportation · doi:10.2514/1.d0364 · Cited by 5 · see citing works
Self-citationApplications and Challenges of Augmente Reality in Education Sector: A Report
Pal, R. A. B., & Ade, M. (2022) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2022.45183 · Cited by 4 · see citing works
Pedagogical Innovations in Personalized Learning
Lahiassi, J., Elwarraki, O., Aammou, S., & Jdidou, Y. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1206-3.ch015 · Cited by 4 · see citing works
Kim, J. W., Wang, K., Miller, J. R., Dorneich, M. C., Winer, E. H., & Brown, L. J. (2023) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc58513.2023.10311255 · Cited by 3 · see citing works
Self-citationKim, J. W., Dorneich, M. C., Winer, E. H., & Wang, K. (2024) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813241262317 · Cited by 2 · see citing works
Self-citationEvaluation of Extended Reality Authoring Tool for General Aviation Weather Training
Kim, J. W., Wang, K., Dorneich, M. C., Winer, E. H., Brown, L. J., & Whitehurst, G. (2024) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc62030.2024.10748866
Self-citationNTSB review of low ceiling/visibility accidents
Eick (2022)
UnresolvedAr lab: Augmented reality app for chemistry education
da Silva (n.d.)
UnresolvedBlippbuilder studio
(2024)
UnresolvedAR media studio
(2024)
UnresolvedVuforia engine
(2024)
UnresolvedARKit
(2024)
UnresolvedARCore
(2024)
UnresolvedWikitude AR SDK
(2024)
UnresolvedTask-centered user interface design
Lewis (1993) · A Practical Introduction
UnresolvedAn XR authoring tool for customizing aviation weather educational content
Wang (n.d.)
UnresolvedThe one number you need to grow
Reichheld (2003) · Harvard Bus. Rev.
UnresolvedWhat is a good net promoter score?
(2024)
UnresolvedRubrics for evaluating open education resource (OER) objects
(2024)
Unresolveduality online course review
(2024)
UnresolvedAviation Weather with Augmented Reality: An Interactive Print Experience For Pilots
Brown (2021)
UnresolvedPrivate Pilot FAA Knowledge Test Prep2021 Edition
Gleim (2020)
UnresolvedISO 9241-11: Ergonomic of human-system interaction Part 11: Usability: Definitions and concepts
(1998)
Unresolved
Flagged first, then by citations
17 of the 59 references couldn’t be matched to a known paper.
Cited by
Still being checked.
- Authors
- Kim, J. W., Wang, K., Dorneich, M. C., Winer, E., Brown, L. J., & Whitehurst, G.
- Affiliations
- Iowa State University (US)
- College of Aviation, Kalamazoo, MI, USA
- Venue
- IEEE Transactions on Human-Machine Systems
- Published
- 2026-06
- Volume
- 56
- Issue
- 3
- Pages
- 572-581
- ISSN
- 2168-2291 (ISSN-L)
- 2168-2305
- OpenAlex
- W7134980363
- Semantic Scholar
- 8c6d1ce0b6107d0fc88b682fed9f3cb274a2f54f
- Cited by
- 0
- References
- 59
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Human-Machine Systems (publishedVersion)
- Iowa State University Digital Repository (Iowa State University) (submittedVersion)
- Funders
- Partnership to Enhance General Aviation Safety, Accessibility, and Sustainability
- Federal Aviation Administration (12-C-GA-ISU)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Augmented Reality Applications
- Physical Sciences › Computer Science › Information Systems › Multimedia Learning Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Keywords
- educational augmented reality, flight instructors, usability, teaching quality, learning outcomes, student engagement, non-technical users, AR in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review