Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant
The DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Abstract
Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Educating end users to spot what could potentially be AI-generated will empower them to establish an understanding of AI tools [67,109] and develop individual-specifc strategies to evolve to be more informed and mindful information consumers.”
From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [109]
In the citing paper
“prior CHI studies [43, 50, 77, 79, 93] have investigated potential scenarios of interacting with multi-party based chatbots [77] as well as diverse roles of chatbots and their positive impact on familiarization among teammates [79], generating more ideas and diverse views [43, 50, 79], and nudging silent people to speak up [43].”
From The Promise and Peril of ChatGPT in Higher Education: Opportunities, Challenges, and Design Implications (2024) · cites this paper as [93]
In the citing paper
“Design guidelines can be difficult to put into practice [59, 113, 160, 164, 176, 192], often because they describe goals rather than actions [73].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [176]
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Word embeddings represent words as vectors in a higher dimensional latent space, where lexico-semantically similar words tend to have vectors that are closer [21, 65, 74].”
BackgroundThe Presentation of Self in Everyday Life.
Naegele, K. D., & Goffman, E. (1956) · American Sociological Review · doi:10.2307/2089106 · Cited by 25,991
Does the autistic child have a “theory of mind” ?
Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985) · Cognition · doi:10.1016/0010-0277(85)90022-8 · Cited by 8,599 · cited 8×
“Scholars posit that the Theory of Mind (ToM) is a basic cognitive and social characteristic that enables us to make conjectures about each others’ minds through observable or latent behavioral and verbal cues [6, 12, 37, 38, 94].”
BackgroundInfluentialVADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087 · cited 1×
“To measure the sentiment of each post to JW, we used the VADER sentiment analysis model [48], which is a rule-based sentiment analysis model that provides numerical scores ranging from -1”
MethodPresentation of Self in Everyday Life, the
(2008) · Encyclopedia of Political Communication · doi:10.4135/9781412953993.n530 · Cited by 3,771
Mindblindness: An Essay on Autism and Theory of Mind
(1995) · Choice Reviews Online · doi:10.5860/choice.33-2412 · Cited by 3,670 · cited 2×
“For example, building shared plans and goals is fundamental to collaborative task-completion— ToM enables us to recognize and mitigate each other’s plans and goals to work together [5, 6]; intentional communication is the basis for smooth communication— ToM enables us to understand the interlocutor has a belief or kno…”
BackgroundThe Uncanny Valley [From the Field]
Mori, M., MacDorman, K., & Kageki, N. (2012) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2192811 · Cited by 3,542 · cited 1×
“Anthropomorphism is a widely studied yet highly debatable design characteristic of CA—on one hand, intentionally building CAs with more humanlike attributes can improve user trust[13, 32], make the CAmore approachable and ease user interactions [54, 97]; on the other hand, the famous “Uncanny Valley” effect [66] indi-”
BackgroundMindblindness: An Essay on Autism and Theory of Mind
Baron-Cohen, S. (1995) · doi:10.7551/mitpress/4635.001.0001 · Cited by 3,428
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 6×
“Even though building artifcially “intelligent” machines has been a unfulflled promise due to various technical and feasibility challenges [8, 87], users tend to expect their CAs to be “smart” [98], thus creating a gap between user expectation and CA’s true intelligence.”
BackgroundMethodInfluentialGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 2×
“While CAs are relatively successful in task-oriented interactions [82, 96], the initial promise of building CAs that can carry out natural and coherent conversations with users has largely remained unfulflled due to both design and technical challenges [3, 18, 87].”
Background
Research works printed but not linked: 14
- Simon Baron-cohen. 1999. Evolution of a Theory of Mind?In The Descent of Mind: Psychological Perspectives on Hominid Evolution. Oxford University Press, 1–31.
- Cristian Danescu-Niculescu-Mizil Moritz Sudhof Dan Jurafsky Jure Leskovec and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078(2013). · arXiv
- Bobbie Eicher, Kathryn Cunningham, Sydni Peterson Marissa Gonzales, and Ashok Goel. 2017. Toward mutual theory of mind as a foundation for co-creation. In International Conference on Computational Creativity, Co-Creation Workshop.
- Ashok K Goel and Lalith Polepeddi. 2016. Jill Watson: A virtual teaching assistant for online education. Technical Report. Georgia Institute of Technology.
- 10.4324/9781003320609-59 · DOI · Paper page
- Alvin I Goldman 2012. Theory of mind. The Oxford handbook of philosophy of cognitive science 1 (2012).
- Kokil Jaidka, Sharath Chandra Guntuku, Anneke Buffone, H Andrew Schwartz, and Lyle H Ungar. 2018. Facebook vs. Twitter: Cross-platform differences in self-disclosure and trait prediction. In Proceedings of the Twelfth International AAAI Conference on Web and Social Media. 141–150.
- 10.5555/1622637.1622649 · doi:10.5555/1622637.1622649
- Tomas Mikolov Ilya Sustskever Kai Chen Greg Corrado and Jeffrey Dean. 2013. Distributed Representations ofWords and Phrases and their Compositionality. In Neural Information Processing Systems (NIPS). 3111–3119.
- Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers.Journal of Social Issues 1, 56 (2000), 81–103.
- Christopher Peters. 2005. Foundations of an agent theory of mind model for conversation initiation in virtual environments. Virtual Social Agents(2005) 163. · DOI
- David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind?Behavioral and brain sciences 1, 4 (1978), 515–526.
- Tina L Robbins and Angelo S DeNisi. 1994. A closer look at interpersonal affect as a distinct influence on cognitive processing in performance evaluations.Journal of Applied Psychology 79, 3 (1994), 341.
- Henry M Wellman. 1992. The child’s theory of mind.The MIT Press.
Cited by
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288
“Educating end users to spot what could potentially be AI-generated will empower them to establish an understanding of AI tools [67, 109] and develop individual-specifc strategies to evolve to be more informed and mindful information consumers.”
BackgroundRoss, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
“Such a pair of models is often described asMutual Theory ofMind [29, 100].”
BackgroundMethodKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“After refning the principles and strategies for relevance and clarity, we evaluated their utility within the design process by conducting structured, exploratory workshops with design practitioners within our organization who work on generative AI applications.”
MethodWhat is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
Metadata
- Authors
- Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-14
- OpenAlex
- W3160674997
- Cited by
- 120
- References
- 96
- Citations by year
- 2026: 20 · 2025: 26 · 2024: 33 · 2023: 26 · 2022: 12 · 2021: 3
- FWCI
- 31.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Georgia Institute of Technology
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- human-AI interaction, virtual teaching assistants, chatbots, student perceptions, anthropomorphism, perceived intelligence, online learning, forum, linguistic features, sentiment analysis, readability, long-term interaction, adaptive agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant. https://app.cochat.ai/papers/10.1145/3411764.3445645
- The paper
- Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021). Towards Mutual Theory of Mind in Human-AI Interaction: How Language Reflects What Students Perceive About a Virtual Teaching Assistant. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445645
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.