CoChat
Sign Up
Cited by 11Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Do students rely on AI? Analysis of student-ChatGPT conversations from a field study.

Open access
Zheng, Jiayu, Hao, Lingxin, Lu, Kelun, Garg, Ashi, Reese, Mike, Yap, Melo-Jean, Wang, I-Jeng, Wu, Xingyun, Huang, Wenrui, Hoffman, Jenna, Kelly, Ariane, Le, My, Zhang, Ryan, Lin, Yanyu, Faayez, Muhammad, Liu, Anqi·2025·Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
DOI 10.1609/aies.v8i3.36760
Open this paper in CoChat — free accountOpen PDF

Abstract

This study explores how college students interact with generative AI (ChatGPT-4) during educational quizzes, focusing on reliance and predictors of AI adoption. Conducted at the early stages of ChatGPT implementation, when students had limited familiarity with the tool, this field study analyzed 315 student-AI conversations during a brief, quiz-based scenario across various STEM courses. A novel four-stage reliance taxonomy was introduced to capture students' reliance patterns, distinguishing AI competence, relevance, adoption, and students' final answer correctness. Three findings emerged. First, students exhibited overall low reliance on AI and many of them could not effectively use AI for learning. Second, negative reliance patterns often persisted across interactions, highlighting students’ difficulty in effectively shifting strategies after unsuccessful initial experiences. Third, certain behavioral metrics strongly predicted AI reliance, highlighting potential behavioral mechanisms to explain AI adoption. The study's findings underline critical implications for ethical AI integration in education and the broader field. It emphasizes the need for enhanced onboarding processes to improve student's familiarity and effective use of AI tools. Furthermore, AI interfaces should be designed with reliance-calibration mechanisms to enhance appropriate reliance. Ultimately, this research advances understanding of AI reliance dynamics, providing foundational insights for ethically sound and cognitively enriching AI practices.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aies.v8i3.36760

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 37

  1. [1]1. Acosta-Enriquez, B. G.; Arbulú Ballesteros, M. A.; Arbulu Perez Vargas, C. G.; Orellana Ulloa, M. N.; Gutiérrez Ul- loa, C. R.; Pizarro Romero, J. M.; Gutiérrez Jaramillo, N. D.; Cuenca Orellana, H. U.; Ayala Anzoátegui, D. X.; and López Roca, C. 2024. Knowledge, attitudes, and perceived Ethics regarding the use of ChatGPT among generation Z university students. International Journal for Educational Integrity, 20(1): 10.
  2. [2]2. Ahmad, S. F.; Han, H.; Alam, M. M.; Rehmat, M.; Irshad, M.; Arraño-Muñoz, M.; Ariza-Montes, A.; et al. 2023. Im- pact of artificial intelligence on human loss in decision mak- ing, laziness and safety in education. Humanities and Social Sciences Communications, 10(1): 1-14.
  3. [3]3. Ammari, T.; Chen, M.; Zaman, S.; and Garimella, K. 2025. How Students (Really) Use ChatGPT: Uncovering Expe- riences Among Undergraduate Students. arXiv preprint arXiv:2505.24126. · arXiv
  4. [4]4. Bansal, G.; Wu, T.; Zhou, J.; Fok, R.; Nushi, B.; Kamar, E.; Ribeiro, M. T.; and Weld, D. 2021. Does the whole ex- ceed its parts? the effect of ai explanations on complemen- tary team performance. In Proceedings of the 2021 CHI conference on human factors in computing systems, 1-16.
  5. [5]5. Bauer, C. F.; Bowe, K. A.; Meredith, D. C.; and Kustina, J. 2025. Students as Partners in Teaching, Managing, and De- veloping Peer Learning in STEM Gateway Courses. Journal of College Science Teaching, 1-9.
  6. [6]6. Cao, S.; Liu, A.; and Huang, C.-M. 2024. Designing for ap- propriate reliance: The roles of ai uncertainty presentation, initial user decision, and user demographics in AI-assisted decision-making. Proceedings of the ACM on Human- Computer Interaction, 8(CSCW1): 1-32.
  7. [7]7. Costa, A. R.; Lima, N.; Viegas, C.; and Caldeira, A. 2024. Critical minds: enhancing education with ChatGPT. Cogent Education, 11(1): 2415286.
  8. [8]8. Dai, Y.; Xiao, J.-Y.; Huang, Y.; Zhai, X.; Wai, F.-C.; and Zhang, M. 2025. How generative AI enables an online project-based learning platform: An applied study of learn- ing behavior analysis in undergraduate students. Applied Sciences, 15(5): 2369.
  9. [9]9. Ding, L.; Li, T.; Jiang, S.; and Gapud, A. 2023. Students' perceptions of using ChatGPT in a physics class as a virtual tutor. International Journal of Educational Technology in Higher Education, 20(1): 63.
  10. [11]11. Fan, Y.; Tang, L.; Le, H.; Shen, K.; Tan, S.; Zhao, Y.; Shen, Y.; Li, X.; and Gašević, D. 2025. Beware of metacognitive laziness: Effects of generative artificial intelligence on learn- ing motivation, processes, and performance. British Journal of Educational Technology, 56(2): 489-530.
  11. [12]12. Fırat, M. 2025. A global triangulation approach to ChatGPT integration in education. Educational Technology Quarterly.
  12. [13]13. Galdames, I. S. 2024. Impact of Artificial Intelligence on Higher Education: A Literature Review. In International Conference in Information Technology and Education, 373- 392. Springer.
  13. [14]14. Gerlich, M. 2025. AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1): 6.
  14. [15]15. Grassini, S. 2023. Shaping the future of education: Explor- ing the potential and consequences of AI and ChatGPT in educational settings. Education sciences, 13(7): 692.
  15. [16]16. Hakiki, M.; Fadli, R.; Samala, A. D.; Fricticarani, A.; Dayurni, P.; Rahmadani, K.; Astiti, A. D.; and Sabir, A. 2023. Exploring the impact of using Chat-GPT on student learning outcomes in technology learning: The comprehen- sive experiment. Advances in Mobile Learning Educational Research, 3(2): 859-872.
  16. [18]18. Hasanein, A. M.; and Sobaih, A. E. E. 2023. Drivers and consequences of ChatGPT use in higher education: Key stakeholder perspectives. European journal of investigation in health, psychology and education, 13(11): 2599-2614.
  17. [19]19. Kahr, P. K.; Rooks, G.; Willemsen, M. C.; and Snijders, C. C. 2024. Understanding trust and reliance development in ai advice: Assessing model accuracy, model explanations, and experiences from previous interactions. ACM Transac- tions on Interactive Intelligent Systems, 14(4): 1-30.
  18. [21]21. Kosar, T.; Ostojić, D.; Liu, Y. D.; and Mernik, M. 2024. Computer science education in chatgpt era: Experiences from an experiment in a programming course for novice pro- grammers. Mathematics, 12(5): 629.
  19. [22]22. Krathwohl, D. R. 2002. A revision of Bloom's taxonomy: An overview. Theory into practice, 41(4): 212-218.
  20. [23]23. Krupp, L.; Steinert, S.; Kiefer-Emmanouilidis, M.; Avila, K. E.; Lukowicz, P.; Kuhn, J.; Küchemann, S.; and Karolus, J. 2024. Unreflected acceptance-investigating the negative consequences of chatgpt-assisted problem solving in physics education. In HHAI 2024: Hybrid Human AI Systems for the Social Good, 199-212. IOS Press.
  21. [24]24. Küper, A.; Lodde, G. C.; Livingstone, E.; Schadendorf, D.; and Krämer, N. 2025. Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Sup- port Systems: Experimental Web-Based Study Among Der- matologists. Journal of Medical Internet Research, 27: e58660.
  22. [25]25. Lai, V.; and Tan, C. 2019. On human predictions with expla- nations and predictions of machine learning models: A case study on deception detection. In Proceedings of the confer- ence on fairness, accountability, and transparency, 29-38.
  23. [27]27. Li, Z.; Lu, Z.; and Yin, M. 2023. Modeling human trust and reliance in ai-assisted decision making: A markovian approach. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 37, 6056-6064.
  24. [28]28. Lundberg, S. M.; and Lee, S.-I. 2017. A unified approach to interpreting model predictions. Advances in neural infor- mation processing systems, 30.
  25. [29]29. McNichols, H.; and Lan, A. 2025. The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelli- gence Course. arXiv preprint arXiv:2503.07928. · arXiv
  26. [31]31. Narayanan, S.; Yu, G.; Ho, C.-J.; and Yin, M. 2023. How does value similarity affect human reliance in AI-assisted ethical decision making? In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 49-57.
  27. [33]33. Paladines, J.; and Ramirez, J. 2020. A systematic literature review of intelligent tutoring systems with dialogue in natu- ral language. IEEE Access, 8: 164246-164267.
  28. [34]34. Santos, P.; Urgel, K.; and Moreno, V. 2024. Generative Arti- ficial Intelligence in teaching and learning of ICT engineer- ing education: A literature review and illustrative scenar- ios. In 2024 47th MIPRO ICT and Electronics Convention (MIPRO), 1338-1343. IEEE.
  29. [36]36. Schemmer, M.; Kuehl, N.; Benz, C.; Bartos, A.; and Satzger, G. 2023. Appropriate reliance on AI advice: Conceptual- ization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Inter- faces, 410-422.
  30. [37]37. Singh, H.; Tayarani-Najaran, M.-H.; and Yaqoob, M. 2023. Exploring computer science students' perception of Chat- GPT in higher education: A descriptive and correlation study. Education Sciences, 13(9): 924.
  31. [38]38. Solomon, D. H.; Jones, S.; Brinberg, M.; Bodie, G. D.; and Ram, N. 2022. Using sequence analysis to identify conver- sational motifs in supportive interactions. Journal of social and Personal Relationships, 39(10): 3155-3179.
  32. [39]39. Sun, T.; Zhao, K.; and Chen, M. 2024. Human-AI Interac- tion: Human Behavior Routineness Shapes AI Performance. IEEE Transactions on Knowledge and Data Engineering.
  33. [41]41. Wang, S.; Wang, F.; Zhu, Z.; Wang, J.; Tran, T.; and Du, Z. 2024. Artificial intelligence in education: A systematic literature review. Expert Systems with Applications, 252: 124167.
  34. [42]42. Wang, Y.; Pan, Y.; Yan, M.; Su, Z.; and Luan, T. H. 2023. A survey on ChatGPT: AI-generated contents, challenges, and solutions. IEEE Open Journal of the Computer Society, 4: 280-302.
  35. [43]43. Wu, F.; Dang, Y.; and Li, M. 2025. A Systematic Review of Responses, Attitudes, and Utilization Behaviors on Gen- erative AI for Teaching and Learning in Higher Education. Behavioral Sciences, 15(4): 467.
  36. [44]44. Yin, M.; Wortman Vaughan, J.; and Wallach, H. 2019. Un- derstanding the effect of accuracy on trust in machine learn- ing models. In Proceedings of the 2019 chi conference on human factors in computing systems, 1-12.
  37. [45]45. Zhai, C.; Wibowo, S.; and Li, L. D. 2024. The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review. Smart Learning Environments, 11(1): 28.

Cited by

Sorted by citations

Metadata

Authors
Zheng, J., Hao, L., Lu, K., Garg, A., Reese, M., Yap, M.-J., Wang, I.-J., Wu, X., Huang, W., Hoffman, J., Kelly, A., Le, M., Zhang, R., Lin, Y., Faayez, M., & Liu, A.
Affiliations
  • Johns Hopkins University (US)
  • University of Pennsylvania (US)
ORCID
Venue
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2025-10-15
Volume
8
Issue
3
Pages
2796-2807
ISSN
3065-8365 (ISSN-L)
DOI
10.1609/aies.v8i3.36760
OpenAlex
W4415195223
Cited by
11
References
5
Citations by year
2026: 11
FWCI
12.58
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI/ACM Conference on AI Ethics and Society (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
generative AI in education, ChatGPT-4, student reliance on AI, artificial intelligence adoption, student-AI interaction, educational quizzes, STEM courses, AI interface design, AI ethics in education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review