A DEMATEL-ANP-Based Evaluation of AI-Assisted Learning in Higher Education
Open accessAbstract
This study proposes an indicator system for evaluating AI-assisted learning in higher education, combining evidence-based indicator development with expert-validated weighting. First, we review recent studies to extract candidate indicators and organize them into coherent dimensions. Next, a Delphi session with domain experts refines the second-order indicators and produces a measurable, non-redundant, implementation-ready index system. To capture interdependencies among indicators, we apply a hybrid Decision-Making Trial and Evaluation Laboratory–Analytic Network Process (DEMATEL–ANP, DANP) approach to derive global indicator weights. The framework is empirically illustrated through a course-level application to examine its decision usefulness, interpretability, and face validity based on expert evaluations and structured feedback from academic staff. The results indicate that pedagogical content quality, adaptivity (especially difficulty adjustment), formative feedback quality, and learner engagement act as key drivers in the evaluation network, while ethics-related indicators operate primarily as enabling constraints. The proposed framework provides a transparent and scalable tool for quality assurance in AI-assisted higher education, supporting instructional design, accreditation reporting, and continuous improvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The rapid emergence of generative AI and large language models after 2022 fundamentally shifted the research landscape, introducing both new opportunities and unprecedented challenges.
“However, the rapid emergence of generative AI and large language models after 2022 fundamentally shifted the research landscape, introducing both new opportunities and unprecedented challenges (Ilieva et al., 2026).”
From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Ilieva et al., 2026
The Introduction (page not provided) says recent generative AI is rapidly transforming higher education, creating new capabilities alongside technical and ethical challenges. It does not establish a shift in the broader research landscape specifically after 2022, or characterize the challenges as unprecedented. (LLM, conf=0.83)
Experimental — accuracy not yet measuredFollowing 2022, research on generative AI and large language models increased markedly, and ChatGPT was the tool most frequently examined in every application category.
“Nevertheless, the period following 2022 saw a marked increase in research on generative AI and large language models, and ChatGPT was the tool most frequently examined in every application category (Ilieva et al., 2026).”
From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Ilieva et al., 2026
The Introduction discusses recent generative AI in higher education, but does not report a post-2022 increase in research. Searches for the post-2022 trend and for ChatGPT found no passage comparing how frequently tools were examined across application categories; the paper’s empirical application evaluates courses instead. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 18
- Preprints.org (www.preprints.org) | NOT PEER-REVIEWED | Posted: 30 December 2025 doi:10.20944/preprints202512.2661.v1 · DOI
- Jdidou, Y.; Aammou, S. Transforming Education with AI: The Role of Recommendation Systems in Personalized Learning. In Proceedings of EDULEARN24: International Conference on Education and New Learning Technologies, Palma de Mallorca, Spain, 1-3 July 2024; pp. 7662-7669. https://doi.org/10.21125/edulearn.2024.1801. · DOI
- Chng, L.K. How AI Makes Its Mark on Instructional Design. Asian J. Distance Educ. 2023, 18, 32-41. Available online: http://asianjde.com/ojs/index.php/AsianJDE/article/view/740 (accessed on 25 December 2025).
- Li, Y.; Xie, M. Navigating International Challenges of Quality Assurance in Higher Education: A Synergy of Gen-AI and Human-Made Solutions. GBP Proc. Ser. 2025, 3, 186-192. doi:10.71222/bk845x09. Available online: https://www.gbspress.com/index.php/GBPPS/article/view/177 (accessed on 9 December 2025). · DOI
- Basri, W.S. Effectiveness of AI-Powered Tutoring Systems in Enhancing Learning Outcomes. Eurasian J. Educ. Res. 2024, 110, 33-52. https://doi.org/10.14689/ejer.2024.110.003. · DOI
- Kakhkharova, M.; Tuychieva, S. AI-Enhanced Pedagogy in Higher Education: Redefining Teaching- Learning Paradigms. In Proceedings of the 2024 International Conference on Knowledge Engineering and Communication Systems (ICKECS), Chikkaballapur, India, 18-19 April 2024; pp. 1-6.
- Li, Z.; Dhruv, A.; Jain, V. Ethical Considerations in the Use of AI for Higher Education: A Comprehensive Guide. In Proceedings of the 2024 IEEE 18th International Conference on Semantic Computing (ICSC), Laguna Hills, CA, USA, 5-7 February 2024; pp. 218-223.
- Gabus, A.; Fontela, E. World Problems, an Invitation to Further Thought within the Framework of DEMATEL; Battelle Geneva Research Center: Geneva, Switzerland, 1972; pp. 12-14.
- The ELEVATE-AI LLMs Framework: An Evaluation Framework for Use of Large Language Models in HEOR: An ISPOR Working Group Report. Available online: Preprints.org (www.preprints.org) | NOT PEER-REVIEWED | Posted: 30 December 2025 doi:10.20944/preprints202512.2661.v1 https://warwick.ac.uk/fac/cross_fac/eduport/edufund/projects/yang/projects/the-elevate-ai-llms- framework-an-evaluation-framework-for-use-of-large-language-models-in-a16e6d/ (accessed on 9 December 2025). ArXiv version. Available online: https://arxiv.org/pdf/2501.12394v1 (accessed on 9 December 2025). · DOI
- Masoumi, D.; Lindström, B. Quality in E-Learning: A Framework for Promoting and Assuring Quality in Virtual Institutions. J. Comput. Assist. Learn. 2012, 28, 27-41. https://doi.org/10.1111/j.1365-2729.2011.00440.x. · DOI
- Dazeley, R.; Goriss-Hunter, A.; Meredith, G.; Sellings, P.; Firmin, S.; Burke, J.; Panther, B. Agile Backward Design: A Framework for Planning Higher Education Curriculum. Aust. Educ. Res. 2025, 52, 1489-1508. https://doi.org/10.1007/s13384-024-00772-7. · DOI
- Branson, R.K. The Interservice Procedures for Instructional Systems Development. Educ. Technol. 1978, 18, 11-14. Available online: https://www.jstor.org/stable/44418942 (accessed on 25 December 2025).
- Blundell, C.N.; Mukherjee, M.; Nykvist, S. A Scoping Review of the Application of the SAMR Model in Research. Comput. Educ. Open 2022, 3, 100093.
- Myrick, J. Integrating AI into Course Quality Assurance Processes. In Proceedings of the 8th International Academic Conference on Education, Teaching and Learning (IACETL), Helsinki, Finland, 10-12 July 2025. Available online: https://www.dpublication.com/wp-content/uploads/2025/06/19-4196.pdf (accessed on 25 December 2025).
- Logan-Fleming, D.; Thakkar, M.; Martinez Pacheco, S.; Jones, H.; Parrish, D. Assurance of Learning in the Age of AI: A Sector Scan and Future-Focused Call to Action. In ASCILITE 2025 Conference Proceedings, Adelaide, Australia, 30 November-3 December 2025.
- Yu, H.; Guo, Y. Generative Artificial Intelligence Empowers Educational Reform: Current Status, Issues, and Prospects. Front. Educ. 2023, 8, 1183162. https://doi.org/10.3389/feduc.2023.1183162. Preprints.org (www.preprints.org) | NOT PEER-REVIEWED | Posted: 30 December 2025 doi:10.20944/preprints202512.2661.v1 · DOI
- Chng, L.K. How AI Makes Its Mark on Instructional Design. Asian J. Distance Educ. 2023, 18, 32-41. Available online: https://asianjde.com/ojs/index.php/AsianJDE/article/view/740 (accessed on 25 December 2025).
- Naseer, F.; Khan, M.N.; Tahir, M.; Addas, A.; Aejaz, S.M.H. Integrating Deep Learning Techniques for Personalized Learning Pathways in Higher Education. Heliyon 2024, 10, e32628. https://doi.org/10.1016/j.heliyon.2024.e32628 Disclaimer/Publisher's Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. Preprints.org (www.preprints.org) | NOT PEER-REVIEWED | Posted: 30 December 2025 doi:10.20944/preprints202512.2661. · DOI
Cited by
Sorted by citations
Shang, X., & Tang, A. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1792669 · Cited by 1 · see citing works
Vélez, W. A. Z., Cobeña, E. I. D., Santana, C. L. M., & Mero, C. A. L. (2026) · Education Sciences · doi:10.3390/educsci16071066 · Cited by 1 · see citing works
Akter, F., & Parvatha Reddy, K. K. (2025) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2025.v07i06.84921
“Nevertheless, the period following 2022 saw a marked increase in research on generative AI and large language models, and ChatGPT was the tool most frequently examined in every application category (Ilieva et al. , 2026).”
Sabardi, W., Syairudin, B., & Gunarta, I. K. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666728
- Authors
- Ilieva, G., Yankova, T., Ruseva, M., & Klisarova-Belcheva, S.
- Affiliations
- Plovdiv University (BG)
- Corresponding author
- Yankova, Tania
- Venue
- Computers
- Publisher
- MDPI AG
- Published
- 2026-02-01
- Volume
- 15
- Issue
- 2
- Pages
- 79
- ISSN
- 2073-431X (ISSN-L)
- OpenAlex
- W7126420943
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 3
- FWCI
- 25.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Plovdiv University "Paisii Hilendarski"
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Educational Technology and Pedagogy
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- AI-assisted learning, higher education, evaluation metrics, indicator system, DEMATEL-ANP, expert validation, indicator weighting, adaptive learning, formative feedback, student engagement, AI ethics, quality assurance, accreditation, instructional design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
