Deception detection in educational AI: challenges for Japanese middle school students in interacting with generative AI robots
Open accessAbstract
Educational materials that utilize generative AI (e.g., ChatGPT) have been developed, thus, allowing students to learn through conversations with robots or agents. However, if these artificial entities provide incorrect information (hallucinating), it could lead to confusion among students. To investigate whether students can detect lies from these artificial entities, we conducted an experiment using the social robot Furhat and we make it engage in various types of deceptive interactions. Twenty-two Japanese middle school students participated in ten teaching sessions with Furhat using a human and an anime facial appearances while employing different types of deception: Lying, Paltering, Pandering, and Bullshit. The results revealed that the majority of students were deceived by those lies. Additionally, the robot's facial appearance (i.e., social agency) affected both the learning effectiveness and the likelihood of being deceived. We conclude that an anime robot face is recommended to be used as it excelled in learning effectiveness as it attracts students attention. An anime face also provided protection against deceptive techniques due to its low social agency which leads to ineffectiveness in persuasion and deception. This study underscores the importance of preparing AI-based educational tools and scripts carefully to prevent the dissemination of false information produced through generative AI hallucinations to students.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI literacy and ethical reasoning objectives in the reviewed studies develop students' understanding of how AI works, their ability to assess truthfulness, detect bias, and recognize the limits of creativity.
“This objective develops students' understanding of how AI works, their ability to assess truthfulness, detect bias, and recognize the limits of creativity.”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
The Results analysis (§4.3 and §4.8, page not provided) examines whether students believed or questioned a robot's statements, supporting the truthfulness-assessment aspect. The paper does not show that an educational objective developed those abilities, nor does it address students learning how AI works, detecting bias, or recognizing limits of creativity. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSeveral studies require students to identify incorrect or fabricated AI outputs, which cultivates critical thinking and information discrimination skills.
“Several studies also require students to identify incorrect or fabricated AI outputs, which cultivates critical thinking and information discrimination skills [100,101,133].”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
Section 3.6.3 (page unavailable) shows that students were asked whether the robot was telling the truth, supporting the part about evaluating deceptive AI output. But Section 4.3 reports that most students believed the robot and did not question faulty content; the paper does not show that the exercise cultivated critical thinking or information-discrimination skills. (LLM, conf=0.87)
Experimental — accuracy not yet measuredAI literacy and ethical reasoning objectives address deficits in technical understanding and resistance to deception.
“It addresses deficits in technical understanding and resistance to deception [132-134].”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
Section 4.3 reports that most participating students believed the robot’s deceptive content, supporting the concern about resistance to deception. The Introduction recommends safeguards against false information, but the paper does not establish a deficit in students’ technical understanding or show that AI-literacy and ethical-reasoning objectives address either deficit. (LLM, conf=0.82)
Experimental — accuracy not yet measuredStudies document limited understanding of model training and datasets, which impairs students' ability to detect errors or deceptive content and leads to weak knowledge/analysis or no significant improvement.
“Studies document limited understanding of model training and datasets [112,122,135,136], which impairs students' ability to detect errors or deceptive content [105,133] and leads to weak knowledge/analysis [109] or no significant improvement [98].”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
The Discussion (section 5; page unavailable) reports that Japanese middle school students believed the robot’s lies across the deception techniques, supporting the claim that students had difficulty detecting deceptive content. The paper does not establish that limited understanding of model training or datasets caused that difficulty. The claims about weak knowledge or no significant improvement are attached to other citations. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThese risks have multiple sources: uneven feedback quality, interaction designs that induce excessive cognitive load, and technical constraints (e.g., imperfect speech recognition, misleading 'hallucinations').
“These risks have multiple sources: uneven feedback quality [124], interaction designs that induce excessive cognitive load [84], and technical constraints (e.g., imperfect speech recognition, misleading 'hallucinations') [78,133].”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
The Introduction identifies generative-AI hallucinations as a source of inaccurate or fabricated information that can mislead students. The paper does not identify imperfect speech recognition as a risk: Section 3.4 describes the robot as performing speech recognition, while Section 4.8 reports complaints about the clarity of its spoken output, not its recognition of speech. (LLM, conf=0.85)
Experimental — accuracy not yet measured
In a third configuration, AI functions as a materials planner, mass-producing concept maps, leveled texts, or gamified problem sets and predefining reading order and pacing, with classroom control largely ceded to AI after teacher screening.
“In a third configuration, AI functions as a materials planner. Systems mass-produce concept maps, leveled texts, or gamified problem sets and predefine reading order and pacing; after teachers screen and correct these resources, classroom control is largely ceded to AI.”
From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [133]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Guest, G., MacQueen, K., & Namey, E. (2012) · doi:10.4135/9781483384436 · Cited by 7,924 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
On seeing human: A three-factor theory of anthropomorphism
Epley, N., Waytz, A., & Cacioppo, J. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.864 · Cited by 4,069 · see citing works
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · see citing works
Interaction Design: Beyond Human-Computer Interaction
Rogers, Y., Sharp, H., & Preece, J. J. (2002) · Open Research Online - ORO (The Open University) · Cited by 1,843 · see citing works
Frankfurt, H. G. (2005) · doi:10.1515/9781400826537 · Cited by 1,175 · see citing works
When We Need A Human: Motivational Determinants of Anthropomorphism
Nicholas Epley, A., Waytz, A. G., Akalis, S. A., & Cacioppo, J. T. (2008) · Social Cognition · doi:10.1521/soco.2008.26.2.143 · Cited by 811 · see citing works
Lying: Moral Choice in Public and Private Life
Rank, H., Bok, S., & Kaufman, H. (1979) · College English · doi:10.2307/376536 · Cited by 702 · see citing works
Nass, C., Fogg, B. J., & Moon, Y. (1996) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1996.0073 · Cited by 665 · see citing works
Baby Schema in Infant Faces Induces Cuteness Perception and Motivation for Caretaking in Adults
Glocker, M. L., Langleben, D. D., Ruparel, K., Loughead, J., Gur, R. C., & Sachser, N. (2009) · Ethology · doi:10.1111/j.1439-0310.2008.01603.x · Cited by 583 · see citing works
Ecological Behavior, Environmental Attitude, and Feelings of Responsibility for the Environment
Kaiser, F. G., Ranney, M. A., Hartig, T., & Bowler, P. A. (1999) · European Psychologist · doi:10.1027//1016-9040.4.2.59 · Cited by 419 · see citing works
Carson, T. L. (2010) · doi:10.1093/acprof:oso/9780199577415.001.0001 · Cited by 400 · see citing works
The effects of feedback on energy conservation: A meta-analysis.
Karlin, B., Zinger, J. F., & Ford, R. (2015) · Psychological Bulletin · doi:10.1037/a0039650 · Cited by 361 · see citing works
Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction
Moubayed, S. A., Beskow, J., Skantze, G., & Granström, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34584-5_9 · Cited by 297 · see citing works
Should we welcome robot teachers?
SHARKEY, A. J. C. (2016) · Ethics and Information Technology · doi:10.1007/s10676-016-9387-z · Cited by 250 · see citing works
Effects of Anthropomorphism and Accountability on Trust in Human Robot Interaction
Natarajan, M., & Gombolay, M. (2020) · Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3319502.3374839 · Cited by 248 · see citing works
Baby schema in human and animal faces induces cuteness perception and gaze allocation in children
Borgi, M., Cogliati-Dezza, I., Brelsford, V., Meints, K., & Cirulli, F. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00411 · Cited by 243 · see citing works
AI Hallucinations: A Misnomer Worth Clarifying
Maleki, N., Padmanabhan, B., & Dutta, K. (2024) · 2024 IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00033 · Cited by 194 · see citing works
We need to talk about deception in social robotics!
Sharkey, A., & Sharkey, N. (2021) · Ethics and Information Technology · doi:10.1007/s10676-020-09573-9 · Cited by 179 · see citing works
Edwards, A., Edwards, C., Spence, P. R., Harris, C., & Gambino, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.06.005 · Cited by 174 · see citing works
Artful Paltering: The Risks and Rewards of Using Truthful Statements to Mislead Others
Rogers, T., Zeckhauser, R., Francesca Gino, B., Norton, M. I., & Schweitzer, M. E. (2014) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000081 · Cited by 170 · see citing works
The case of classroom robots: teachers’ deliberations on the ethical tensions
Serholt, S., Barendregt, W., Vasalou, A., Alves‐Oliveira, P., Jones, A., Petisca, S., & Paiva, A. (2016) · AI & Society · doi:10.1007/s00146-016-0667-2 · Cited by 165 · see citing works
Kwak, S. S., Kim, Y., Kim, E., Shin, C., & Cho, K. (2013) · IEEE RO-MAN · doi:10.1109/roman.2013.6628441 · Cited by 155 · see citing works
Yang, Q.-F., Lian, L.-W., & Zhao, J.-H. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00391-9 · Cited by 152 · see citing works
Kim, J., Merrill, K., Xu, K., & Kelly, S. E. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107383 · Cited by 147 · see citing works
Henderson, A. (2002) · Ubiquity · doi:10.1145/512526.512528 · Cited by 134 · see citing works
The Effectiveness of Educational Robots in Improving Learning Outcomes: A Meta-Analysis
Wang, K., Sang, G., Huang, L.-Z., Li, S., & Guo, J.-W. (2023) · Sustainability · doi:10.3390/su15054637 · Cited by 102 · see citing works
Brink, K. A., & Wellman, H. M. (2020) · Developmental Psychology · doi:10.1037/dev0000884 · Cited by 89 · see citing works
Schauer, F. F., & Zeckhauser, R. (2009) · Stanford University Press eBooks · doi:10.1515/9781503626607-004 · Cited by 87 · see citing works
Emotional Storytelling Using Virtual and Robotic Agents
Costa, S., Brunete, A., Bae, B.-C., & Mavridis, N. (2018) · International Journal of Humanoid Robotics · doi:10.1142/s0219843618500068 · Cited by 83 · see citing works
Klarin, J., Hoff, E. V., Larsson, A., & Daukantaité, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1415782 · Cited by 74 · see citing works
Dou, X., Wu, C., Lin, K., Gan, S., & Tseng, T. (2020) · International Journal of Social Robotics · doi:10.1007/s12369-020-00654-9 · Cited by 71 · see citing works
Informing the design of a virtual environment to support learning in children
Scaife, M., & Rogers, Y. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2001.0473 · Cited by 58 · see citing works
Technology and Behavioral Design in Tourism
Tussyadiah, I. P. (2016) · Tourism on the verge · doi:10.1007/978-3-319-42773-7_12 · Cited by 58 · see citing works
Coeckelbergh, M. (2017) · Ethics and Information Technology · doi:10.1007/s10676-017-9441-5 · Cited by 57 · see citing works
Reich-Stiebert, N., & Eyssel, F. (2017) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2909824.3020242 · Cited by 56 · see citing works
Abdi, E., Tojib, D., Seong, A. K., Pamarthi, Y., & Millington-Palmer, G. (2022) · Scientific Reports · doi:10.1038/s41598-022-19501-0 · Cited by 47 · see citing works
Can a Robot Catch You Lying? A Machine Learning System to Detect Lies During Interactions
Gonzalez-Billandon, J., Aroyo, A. M., Tonelli, A., Pasquali, D., Sciutti, A., Gori, M., Sandini, G., & Rea, F. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00064 · Cited by 37 · see citing works
Isaac, A., & Bridewell, W. B. (2017) · Oxford University Press eBooks · doi:10.1093/oso/9780190652951.003.0011 · Cited by 31 · see citing works
Diederich, S., Lichtenberg, S., Brendel, A., & Trang, S. T.-N. (2019) · Journal of the Association for Information Systems · Cited by 25 · see citing works
MacWilliams, M. W. (2014) · Routledge eBooks · doi:10.4324/9781315703152 · Cited by 20 · see citing works
Johannessen, L. E. F., Rasmussen, E. B., & Haldar, M. (2023) · British Journal of Sociology of Education · doi:10.1080/01425692.2023.2203360 · Cited by 18 · see citing works
ChatGPT, an Excellent Liar: How Conversational Agents’ Hallucinations Impact Learning and Teaching
Salamin, A.-D., Russo, D., & Rueger, D. (2023) · The Proceedings of the International Conference on Teaching, Learning and Education. · doi:10.33422/6th.iacetl.2023.11.100 · Cited by 12 · see citing works
Mindreading deception in dialog
Isaac, A., & Bridewell, W. B. (2013) · Cognitive Systems Research · doi:10.1016/j.cogsys.2013.07.001 · Cited by 8 · see citing works
Berrick: a low-cost robotic head platform for human-robot interaction
Berra, R., Setti, F., & Cristani, M. (2019) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2019.8913932 · Cited by 7 · see citing works
Motivating students' learning in science and technology by using robots
Zhang, J., Lai, I. K. W., & Xu, W. (2020) · International Journal of Innovation and Learning · doi:10.1504/ijil.2020.107620 · Cited by 7 · see citing works
Dynamics of Perceptible Agency: The Case of Social Robots
Brincker, M. (2016) · Minds and Machines · doi:10.1007/s11023-016-9405-2 · Cited by 5 · see citing works
Understanding the Effect of Anthropomorphic Design: Towards More Persuasive Conversational Agents
Cui, T., Peng, X., & Wang, X. (2020) · Journal of the Association for Information Systems · Cited by 2 · see citing works
Bibliography of process biochemistry
(1997) · Process Biochemistry · doi:10.1016/s0032-9592(97)90060-1 · Cited by 1 · see citing works
A Comparative Human-Robot Interaction Study between Face-Display and an Advanced Social Robot
Salem, A., & Sumi, K. (2024) · IEEE Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00090 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
2 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Dodge T. (2009). The effects of interactivity and visual realism on children's cognitive empathy toward narrative characters (PhD thesis). Bloomington, IN: Indiana University, ProQuest Dissertations and Theses.
- Keane T. Chalmers C. Williams M. Boden M. (2016). “The impact of humanoid robots on students' computational thinking,” in Australian Council for Computers in Education 2016 Conference: Refereed Proceedings (The Queensland Society for Information Technology in Education (QSITE), Australia), 93– 102.
- Laurel B. (1993). Computers as Theatre. New York, NY: Addison-Wesley.
- Wood G. (2024). The impact of AI literacy training on the perception of anthropomorphism and trust in conversational AI (master thesis). Eindhoven University of Technology, Eindhoven, Netherlands. Available at: https://research.tue.nl/files/335721461/Master_Thesis_Geert_Wood.pdf
Cited by
Sorted by citations
Lin, X., & Tan, H. (2025) · Systems · doi:10.3390/systems13100840 · Cited by 22 · see citing works
Giansanti, D., Lastrucci, A., Iannone, A., & Pirrera, A. (2025) · Applied Sciences · doi:10.3390/app15073793 · Cited by 9 · see citing works
Yeung, R. C. Y., Sun, D., Yeung, C. H., Keane, T., & Lee, J. C. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105659 · Cited by 2 · see citing works
Huang, Q. (2025) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v19i12.56397
Saarathy, S. C. P., & Riaz, S. (2026) · World Symposium on Artificial Intelligence (WSAI) · doi:10.1109/wsai70435.2026.11708621
- Authors
- Salem, A., & Sumi, K.
- Affiliations
- Future University Hakodate (JP)
- Corresponding author
- Sumi, Kaoru
- ORCID
- Sumi, Kaoru
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2024-12-18
- Volume
- 7
- Pages
- 1493348-1493348
- ISSN
- 2624-8212 (ISSN-L)
- PMC
- PMC11688650
- OpenAlex
- W4405541368
- Cited by
- 5
- References
- 52
- Citations by year
- 2026: 2 · 2025: 3
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Artificial Intelligence (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Social Psychology › Deception detection and forensic psychology
- Keywords
- deception detection, AI hallucination, social robots, junior high school students, lying, pandering, bullshit, social agency, persuasion, learning effectiveness, ChatGPT, artificial intelligence in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
