CoChat
Sign Up
Cited by 2Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningOnline Learning and Analytics

GenAI Analytics and Pedagogical Configurations: Featuring New Generative AI Systems in Education

Open access
Ortega-Arranz, Alejandro, Topali, Paraskevi, García-Zarza, Pablo, Bote-Lorenzo, Miguel L., Asensio-Pérez, Juan I., Molenaar, Inge·2026·Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference
DOI 10.1145/3785022.3785054
Open this paper in CoChat — free accountOpen PDF

Abstract

The appearance of general purpose Large Language Models (LLMs) and their derived chatbots (e.g., ChatGPT, Gemini) has revolutionized how students learn in high school and university settings. While these chatbots provide potential benefits to students (e.g., 24/7 tutor), they hinder the teachers’ autonomy, who are blinded to the questions made by the students and hand-tied to the answers provided by these models. However, such autonomy could be increased with the integration of GenAI Analytics tracking the students’ interactions with the chatbots (e.g., which are the most frequently asked topics), and with the configuration and contextualization of such models. This study aims at understanding which GenAI Analytics and Pedagogical Configurations are more useful for university teachers through a GenAI system developed by the authors. The evaluation involved 16 university teachers from 5 different institutions. Results show that analytics such as the number of prompts or the categorization of their contents are of great interest for teachers. Furthermore, while some pedagogical configurations were positively rated by most participants (e.g., not providing the direct solution, customized add-ons), some others were not (e.g., forced hallucinations, minimum answer length). By better understanding these two features of GenAI systems (i.e., GenAI Analytics and Pedagogical Configurations for chatbot behaviors), we expect teachers will increase their autonomy and trust in GenAI-based…

AI snapshot

Written from the paper’s full text.

TL;DR

This study examines which GenAI analytics and pedagogical configurations university teachers consider useful in a chatbot developed by the authors. In a workshop with 16 teachers, prompt monitoring and content tagging were highly valued, as were configurations for handling direct solution requests and contextualizing answers with course files, while several restrictive options were less favored.

Key findings

  • All evaluated GenAI analytics received positive ratings (4 or 5) from more than 75% of participants; tagging and filtering student prompts with teacher-defined tags was the most useful feature and received no negative scores. The repeated-topic alert was useful to 14 participants (87.5%), and the suspicious copy-paste alert to 12 (75%).
  • For pedagogical configurations, configuring behavior when students ask for direct solutions and contextualizing answers with selected course files were each rated useful by 15 participants (93.75%); configuring behavior when answers are not found in those files was rated useful by 13 (81.25%). Minimum answer length, artificial hallucinations, and daily prompt limits were each rated useful by 8 participants (50%).
  • The developed system’s Net Promoter Score was 6.25, compared with -25 for ChatGPT, a difference of 31.25 points. UMUX-Lite usability items scored m=5.06/7 and m=5.87/7, corresponding to a 71.31 SUS score.
  • Participants suggested additional analytics and configurations, including student profiles based on chatbot use, prompt length and quality, repeated questions, answer-quality ratings, alerts for misleading prompts or missing expected tags, pointers to course materials, forbidden off-topic questions, scaffolding steps, and preventing copy-paste of exercises.

Methods

  • A one-hour face-to-face workshop involved 16 university ICT teachers from five institutions; most were male (11 male, 5 female), ages ranged from 31 to 70, and average teaching experience was 18.1 years (SD 11.3).
  • Participants completed a pre-questionnaire on demographics and GenAI adoption, received a 15-minute introduction, then used teacher and student credentials in a Canvas programming course with a chatbot populated with fictional student data; they configured and tested the tool and reviewed its analytics.
  • A post-questionnaire rated each GenAI analytic and pedagogical configuration on a 5-point Likert item with open-ended justification, plus the Net Promoter Score and UMUX-Lite usability questionnaire.

Limitations

  • The authors state that the implemented GenAI analytics are low-level or raw, and suggest future work on higher-level analytics that combine indicators into profiles or engagement measures.
  • The authors state that the sample was limited to practitioners in educational technology, and future work should include teachers from other disciplines and educational levels.
  • The authors state that data privacy and protection need further study because some systems use commercial LLMs that do not store data locally, potentially sharing student analytics with third parties; it is also evident from the text that the evaluation used a one-hour workshop with fictional data rather than a real course deployment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3785022.3785054

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 19 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    When students use GenAI tools, teachers' autonomy is compromised because teachers cannot be aware of the students' use nor align the behavior of the GenAI agents with their pedagogical intentions.

    “In this regard, when students use GenAI tools (i.e., automated systems), teachers' autonomy is compromised as they cannot be aware of the students' use nor align the behavior of the GenAI agents with their pedagogical intentions (e.g.adapting the answers to the course curricula) [14].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [14] 14. Alejandro Ortega-Arranz, Paraskevi Topali, and Inge Molenaar. 2025. Configur- ing and monitoring students' interactions with generative AI tools: Supporting teacher autonomy. In 15th International Learning Analytics and Knowledge Con- ference. 895-902. doi:10.1145/3706468.3706533 source
  • Premisecitation

    Teacher autonomy is related to the limited capability of GenAI systems to incorporate pedagogical decisions and to contextualize their answers.

    “This issue is closely related to teacher autonomy and to the limited capability of these systems to incorporate pedagogical decisions (e.g., adjusting the depth of GenAI answers) and to contextualize their answers (e.g., limiting GenAI feedback to course documentation and curriculum) [3,5].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [3] 3. Mutlu Cukurova. 2025. Promoting and protecting teacher agency in the age of artificial intelligence. United Nations Educational, Scientific and Cultural Organi- zation (UNESCO). https://teachertaskforce.org/knowledge-hub/promoting-and- protecting-teacher-agency-age-artificial-intelligence

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5] 5. Michail Giannakos, Roger Azevedo, Peter Brusilovsky, Mutlu Cukurova, Yannis Dimitriadis, Davinia Hernandez-Leo, Sanna Järvelä, Manolis Mavrikis, and Bart Rienties. 2025. The promise and challenges of generative AI in education. Be- haviour & Information Technology 44, 11 (2025), 2518-2544. doi:10.1080/0144929X. 2024.2394886 source

      Section 4.1 (p. 17) says contextualisation and quality assurance remain the responsibility of human experts, and Section 3.4.1 (p. 10) says context-appropriate learning design may still require human expertise. This supports a general concern about GenAI incorporating pedagogical context, but the paper does not establish the stated link to teacher autonomy or the specific limitations on answer depth and restricting feedback to course documentation and curriculum. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    The Net Promoter Score of the GenAI system was 6.25 (5 detractors, 6 promoters, 5 neutrals).

    “The Net Promoter Score of the GenAI system is 6.25 (5 detractors, 6 promoters, 5 neutrals).”
    Quote in the paperSubstantial claim
  • Findingown data

    The UMUX-LITE usability score for the item 'the system capabilities meet my requirements' was m=5.06/7.

    “Furthermore, the results of the UMUX-LITE questionnaire (usability) showed a high score for both items: m=5.06/7 (the system capabilities meet my requirements) and m=5.87/7 (the system is easy to use).”
    Quote in the paperSubstantial claim
  • Findingown data

    The UMUX-LITE usability score for the item 'the system is easy to use' was m=5.87/7.

    “Furthermore, the results of the UMUX-LITE questionnaire (usability) showed a high score for both items: m=5.06/7 (the system capabilities meet my requirements) and m=5.87/7 (the system is easy to use).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 7

  • Classified as probably not a substantial claim.

    Premisecitation

    Teachers' autonomy depends on allowing teachers to align the behavior of GenAI agents with their pedagogical intentions.

    “and, (2) allowing teachers to align the behavior of GenAI agents with their pedagogical intentions (i.e., including teachers in the”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] 4. Mutlu Cukurova and Fengchun Miao. 2024. AI competency framework for teachers. UNESCO Publishing. doi:10.54675/ZJTE2084 source

      The Teacher Competency section (p. 39) says teachers should be able to customize AI tools to create inclusive learning environments, and Human-AI hybrid coach (p. 41) describes customizing a generative-AI agent. The paper also says teachers should remain accountable for pedagogical decisions (section 2.2, p. 16). It does not state that teachers’ autonomy depends on being able to align GenAI agents’ behavior with their pedagogical intentions. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GenAI Analytics in education was first introduced, although not defined, by Khosravi et al. in a 2023 Journal of Learning Analytics editorial.

    “The concept of”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] 8. Hassan Khosravi, Olga Viberg, Vitomir Kovanovic, and Rebecca Ferguson. 2023. Generative AI and learning analytics. Journal of Learning Analytics 10, 3 (2023), 1-6. doi:10.18608/jla.2023.8333 source

      The editorial’s Abstract calls for robust GenAI analytics, and its Introduction uses the term in discussing educational research directions without giving it a formal definition (page numbers unavailable). Our copy does not establish that Khosravi et al. were the first to introduce the concept. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Khosravi et al. called for more studies to understand which data points, key metrics and methodologies of GenAI Analytics are relevant for research and practice.

    “In such editorial note, authors claim for more studies trying to understand, among other things, which are the data points (e.g., prompts, model parameters, user responses), key metrics (e.g., duration of the interactions, conversation depth, interaction frequency) and methodologies of GenAI Analytics relevant for both research and practice [8].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] 8. Hassan Khosravi, Olga Viberg, Vitomir Kovanovic, and Rebecca Ferguson. 2023. Generative AI and learning analytics. Journal of Learning Analytics 10, 3 (2023), 1-6. doi:10.18608/jla.2023.8333 source

      The Introduction (page unavailable) invites further contributions on GenAI and learning analytics, and the Abstract (page unavailable) calls for robust GenAI analytics relevant to research and practice. The cited paper does not specify the data points, interaction metrics, or methodologies listed in the citing sentence as topics for study. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Khosravi et al. raised the issue of which teaching methodologies can be better employed with GenAI systems to impact individual and group learning outcomes.

    “Regarding pedagogical decisions, Khosravi et al. [8] also raised this issue by questioning which teaching methodologies can be better employed with these systems to impact individual and group learning outcomes.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] 8. Hassan Khosravi, Olga Viberg, Vitomir Kovanovic, and Rebecca Ferguson. 2023. Generative AI and learning analytics. Journal of Learning Analytics 10, 3 (2023), 1-6. doi:10.18608/jla.2023.8333 source

      The Introduction discusses potential educational uses of GenAI and invites further exploration of its role in learning. It does not raise the attributed question about which teaching methodologies work better with GenAI for individual and group learning outcomes (searched for teaching methodologies, pedagogical approaches, and those outcomes). (LLM, conf=0.73)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The possibility of tagging and filtering students' prompts with predefined tags by teachers was identified as the most useful GenAI analytic feature and was not negatively scored by any participant.

    “Among the different analytics, the possibility of tagging and filtering students' prompts with predefined tags by the teachers was identified as the most useful feature, not being negatively scored by any participant (e.g.,”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The three least-rated pedagogical configurations-minimum answer length, artificial hallucinations, and daily prompt limit-were each rated as useful by 8 participants (50%).

    “On the other hand, configuring a minimum length for the answers, adding artificial hallucinations to the given answers and limiting the number of prompts per day are the features least rated.These three configurations were rated as useful by”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The implemented GenAI Analytics are low-level analytics (raw data).

    “First, the implemented GenAI Analytics are low-level analytics (i.e., raw data).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Positive Net Promoter Scores indicate potential for user loyalty and adoption.

    “Positive scores indicate its potential for user loyalty and adoption.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] 15. Frederick F Reichheld. 2003. The one number you need to grow. Harvard business review 81, 12 (2003), 46-55.

  • May combine two claims.

    Interpretationcitation

    The UMUX-LITE scores can be transformed into a 71.31 SUS score, representing a good level of usability.

    “These scores can be transformed into a 71.31 SUS score [9], representing a good level of usability [17] although with some margin for improvement.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [9] 9. James R Lewis, Brian S Utesch, and Deborah E Maher. 2013. UMUX-LITE: when there's no time for the SUS. In Proceedings of the SIGCHI conference on human factors in computing systems. 2099-2102. doi:10.1145/2470654.2481287 source
    • cites [17] 17. Jeff Sauro. 2018. 5 Ways to Interpret a SUS Score. https://measuringu.com/ interpret-sus-score/

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Analyzing student traces with GenAI systems is not new, especially for research purposes.

    “Analyzing the student traces with GenAI systems is not something new, especially for research purposes (e.g., [1,12]).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] 1. Carlos Alario-Hoyos, Rebiha Kemcha, Carlos Delgado Kloos, Patricia Callejo, Iria Estévez-Ayres, David Santín-Cristóbal, Francisco Cruz-Argudo, and José Luis López-Sánchez. 2024. Tailoring Your Code Companion: Leveraging LLMs and RAG to Develop a Chatbot to Support Students in a Programming Course. In IEEE International Conference on Teaching, Assessment and Learning for Engineering. doi:10.1109/TALE62452.2024.10834365 source
    • cites [12] 12. Taeho Min, Bongwoo Lee, and Hunkoog Jho. 2025. Integrating generative artificial intelligence in the design of scientific inquiry for middle school students. Educa- tion and Information Technologies (2025), 1-32. doi:10.1007/s10639-025-13410-1 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The UMUX-Lite questionnaire is a seven-scale two-item questionnaire with high internal reliability, able to predict and compare results obtained with the SUS questionnaire.

    “The usability of the GenAI system through the UMUX-Lite questionnaire [9], a seven-scale two-item questionnaire with a high internal reliability, able to predict and compare results obtained with the System Usability Scale (SUS) questionnaire.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] 9. James R Lewis, Brian S Utesch, and Deborah E Maher. 2013. UMUX-LITE: when there's no time for the SUS. In Proceedings of the SIGCHI conference on human factors in computing systems. 2099-2102. doi:10.1145/2470654.2481287 source

  • May combine two claims.

    Study design and conductnone given

    The study involved a limited sample of practitioners, all within the field of educational technology.

    “Moreover, the study involved a limited sample of practitioners, all of them within the field of educational technology.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Monitoring students' GenAI use and affecting the pedagogical configuration of GenAI chatbots are a way to favor Human-AI Collaboration adapting to the autonomy needed by each teacher.

    “These two factors (i.e., monitoring the students' GenAI use and the possibility of affecting the pedagogical configuration of GenAI chatbots) are a way to favor Human-AI Collaboration adapting to the autonomy needed by each teacher.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Teachers' autonomy depends on allowing teachers to be aware of how their students use GenAI agents.

    “According to the above definition, teachers' autonomy depends on two important factors: (1) allowing teachers to be aware of how their students use GenAI agents [3].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [3] 3. Mutlu Cukurova. 2025. Promoting and protecting teacher agency in the age of artificial intelligence. United Nations Educational, Scientific and Cultural Organi- zation (UNESCO). https://teachertaskforce.org/knowledge-hub/promoting-and- protecting-teacher-agency-age-artificial-intelligence

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    One of the most important concerns of teachers about GenAI-powered systems is their reliability and the possibility of giving decontextualized and/or wrong answers.

    “One of the most important concerns of teachers about GenAIpowered systems is its reliability, and the possibility of giving decontextualized and/or wrong answers [23].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] 23. Xin Zhang, Peng Zhang, Yuan Shen, Min Liu, Qiong Wang, Dragan Gašević, and Yizhou Fan. 2024. A systematic literature review of empirical research on applying generative artificial intelligence in education. Frontiers of Digital Education 1, 3 (2024), 223-245. doi:10.1007/s44366-024-0028-5 source

  • May combine two claims.

    Premisecitation

    Every year, more professionals use GenAI tools in their daily work, delegating some of their tasks to AI.

    “Additionally, every year, more and more professionals (who previously were students) use these tools in their daily work, delegating some of their tasks to the AI [16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] 16. Glassdoor Economic Research. 2024. Conversation Starter: Chat- GPT Usage in the Workplace More than Doubles Year Over Year. https://www.glassdoor.com/blog/conversation-starter-chatgpt-usage-in- the-workplace-more-than-doubles-year-over-year/

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    The use of GenAI systems by students might change the type of information that teachers perceive from them.

    “As said by Łodzikowski et al. [11], the use of GenAI systems by the students might change the type of information that teachers perceive from them.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] 11. Kacper Łodzikowski, Peter W. Foltz, and John T. Behrens. 2024. Generative AI and Its Educational Implications. Springer Nature Switzerland, 35-57. source
17 of 17

  • The paper relies on this without support.

    There is a lack of studies exploring which analytics are important for keeping adequate levels of teacher autonomy.

    “However, there is a lack of studies exploring which analytics are important for keeping adequate levels of teacher autonomy, thus using such information to detect deviations of the GenAI use with respect to their pedagogical intentions (e.g., methodologies, assessments), and to adapt upcoming course sessions to student needs;”

  • The paper relies on this without support.

    There is a lack of studies addressing which are the most useful GenAI analytics supporting the autonomy of higher education teachers in their practice.

    “Nevertheless, while there is an increasing interest in GenAI Analytics for research, there is a lack of studies addressing which are the most useful GenAI analytics supporting the autonomy of higher education teachers in their practice.”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Participant age range excludes two reported participants

    Minor

    The stated range ends at 50, but the same sentence places two participants in the 61-70 age group.

  • Hilliger study has conflicting publication years

    Minor

    The text cites the Hilliger study as a 2025 publication, while reference 6 lists it as published in 2026.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Mutlu Cukurova. 2025. Promoting and protecting teacher agency in the age of artificial intelligence. United Nations Educational, Scientific and Cultural Organization (UNESCO). https://teachertaskforce.org/knowledge-hub/promoting-and-protecting-teacher-agency-age-artificial-intelligence
  2. Jing Huang. 2005. Teacher autonomy in language learning: A review of the research. Research studies in education 3 (2005) 203–218.

Cited by

Sorted by citations

Related

Authors
Ortega-Arranz, A., Topali, P., García-Zarza, P., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., & Molenaar, I.
Affiliations
  • School of Computer Engineering, Universidad de Valladolid, Valladolid, Spain
  • Radboud University Nijmegen (NL)
  • Universidad de Valladolid (ES)
  • School of Telecommunications Engineering, Universidad de Valladolid, Valladolid, Spain
ORCID
Venue
Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference
Publisher
ACM
Published
2026-04-26
Pages
779-785
DOI
10.1145/3785022.3785054
OpenAlex
W7155644156
Cited by
2
References
15
Citations by year
2026: 2
FWCI
20.47
Type
conference paper
Language
en
Open access
Locations
  • International Conference on Learning Analytics and Knowledge (LAK) (publishedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
Funders
  • Spanish State Research Agency (MICIU/AEI/10.13039/501100011033) together with the ERDF (PID2023-146692OB-C32)
  • Regional Government of Castile and Leon together with the FEDER (VA176P23)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
generative AI, AI-driven analytics, educational technology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review