The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Ethical considerations, including transparency, play an important role when using artificial intelligence (AI) in education. Explainable AI has been coined as a solution to provide more insight into the inner workings of AI algorithms. However, carefully designed user studies on how to design explanations for AI in education are still limited. The current study aimed to identify the effect of explanations of an automated essay scoring system on students’ trust and motivation. The explanations were designed using a needs-elicitation study with students in combination with guidelines and frameworks of explainable AI. Two types of explanations were tested: full-text global explanations and an accuracy statement. The results showed that both explanations did not have an effect on student trust or motivation compared to no explanations. Interestingly, the grade provided by the system, and especially the difference between the student’s self-estimated grade and the system grade, showed a large influence. Hence, it is important to consider the effects of the outcome of the system (here: grade) when considering the effect of explanations of AI in education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 2×
“This could also be used to identify how quickly trust decreases and can be rebuilt after the system made an error (cf., Dietvorst et al., 2015; Dzindolet et al., 2003).”
BackgroundMcAuley, E., Duncan, T., & Tammen, V. V. (1989) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1989.10607413 · Cited by 2,694
Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.197 · Cited by 2,517 · cited 1×
“Specifically, need for cognition positively influenced motivation, which is in line with the definition of need for cognition: the tendency to engage in and enjoy effortful cognitive activities (Cacioppo et al., 1996).”
BackgroundFoundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“For example, soundness proved to be important, as poor explanations hinder acceptance (Herlocker et al., 2000).”
BackgroundShin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 2×
“This could also be used to identify how quickly trust decreases and can be rebuilt after the system made an error (cf., Dietvorst et al., 2015; Dzindolet et al., 2003).”
BackgroundDietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230
The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research
Wisniewski, B., Zierer, K., & Hattie, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03087 · Cited by 1,199 · cited 1×
“This is in line with the general idea that feedback is more effective when more information is provided (Hattie, 2008; Wisniewski et al., 2020).”
BackgroundExplainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863 · cited 7×
“Moreover, these explanations are key to enabling students to act upon the system’s decision or score (Khosravi et al., 2022) and could be seen as tools to improve self-regulated learning skills, thus aiding students to act in more self-motivated and empowered ways (cf. Ferguson, 2019; Selwyn, 2019).”
BackgroundMethodInfluentialA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Nazaretsky, T., Ariely, M., Cukurova, M., & Alexandron, G. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13232 · Cited by 589
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“Ashoori and Weisz (2019) found that stating the model confidence had a small effect on user trust, while other studies did not find that providing an accuracy statement increased confidence (Cramer et al., 2008; Yin et al., 2019).”
BackgroundIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471
The effects of transparency on trust in and acceptance of a content-based art recommender
Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-008-9051-3 · Cited by 448
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · cited 1×
“It has been argued that completeness is even more important than soundness for increasing trust (Kulesza et al., 2013).”
BackgroundThe Very Efficient Assessment of Need for Cognition: Developing a Six-Item Version
Lins de Holanda Coelho, G., H. P. Hanel, P., & J. Wolf, L. (2020) · Assessment · doi:10.1177/1073191118793208 · Cited by 358 · cited 1×
“Need for cognition , or one’s tendency to engage in effortful cognitive activity, was measured by the Need for Cognition scale (Lins de Holanda Coelho et al., 2020).”
MethodUnderstanding algorithm aversion: When is advice from automation discounted?
Prahl, A., & Van Swol, L. (2017) · Journal of Forecasting · doi:10.1002/for.2464 · Cited by 308 · cited 1×
“However, even in cases where systems show high accuracy, we see that humans tend to have low trust (Prahl & van Swol, 2017).”
BackgroundWilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
Choi, S., Jang, Y., & Kim, H. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2049145 · Cited by 294 · cited 1×
“Trust, in turn, has been shown to affect the user’s acceptance of AI technology (Choi et al., 2023).”
BackgroundManzey, D., Reichenbach, J., & Onnasch, L. (2012) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343411433844 · Cited by 277
Bringing Transparency Design into Practice
Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., & Hussmann, H. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172961 · Cited by 250 · cited 2×
“Eiband et al. (2018) created a stage-based participatory process in which the content (what to explain) and format (how to explain) are iteratively evaluated with users.”
BackgroundMueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
Understanding User Trust in Artificial Intelligence-Based Educational Systems: Evidence from China
Qin, F., Li, K., & Yan, J. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12994 · Cited by 204
Toolkit to support intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864353 · Cited by 143 · cited 4×
“Similarly, Lim and Dey (2010) developed the Intelligibility Toolkit to help developers apply the right form of explanation for context-aware applications.”
BackgroundMethodInfluentialExplainable Artificial Intelligence: An Introduction to Interpretable Machine Learning
Kamath, U., & Liu, J. (2021) · doi:10.1007/978-3-030-83356-5 · Cited by 126 · cited 1×
“XAI aims to create a toolbox of approaches and techniques to support human agents “to understand how, when, and why predictions are made” (Kamath & Liu, 2021, p. 2), for instance by adding (non-technical) explanations, interpretations, and justifications to machine outputs.”
BackgroundEthical issues in learning analytics: a review of the field
Tzimas, D., & Demetriadis, S. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-021-09977-4 · Cited by 119 · cited 1×
“It has been argued that a lack of transparency will result in lower trust and higher resistance towards AI (Tzimas & Demetriadis, 2021).”
BackgroundThe Measurement of the Propensity to Trust Automation
Jessup, S. A., Schneider, T. R., Alarcon, G. M., Ryan, T. J., & Capiola, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21565-1_32 · Cited by 90 · cited 2×
“Dispositional trust was measured with the Propensity to Trust Technology (PTT) scale (Jessup et al., 2019).”
MethodResultDo You Still Trust Me? Human-Robot Trust Repair Strategies
Esterwood, C., & Robert, L. P. (2021) · 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man50785.2021.9515365 · Cited by 75 · cited 1×
“Additionally, research has studied ways to communicate after a system failure, for instance, by using different repair strategies, apologies, promises, justifications, or explanations with different attributions (cf. Kim & Song, 2021; Esterwood & Robert, 2021; Matzat & Snijders, 2012).”
Background
Cited by
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
“…considering that it is essential to study the importance of the grading explanation and transparency of the grading process that these systems are reporting to the students, which may be a pivotal aspect considering both ethical concerns related to the technology and its acceptability [30,32].”
BackgroundHarnessing AI to Foster Equity in Education: Opportunities, Challenges, and Emerging Strategies
Roshanaei, M., Olivares, H., & Lopez, R. R. (2023) · Journal of Intelligent Learning Systems and Applications · doi:10.4236/jilsa.2023.154009 · Cited by 147
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130
Aljuaid, H. (2024) · Arab World English Journal · doi:10.24093/awej/chatgpt.2 · Cited by 129
Zheng, W., & Cheung Tse, A. W. (2023) · 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) · doi:10.1109/tale56641.2023.10398319 · Cited by 13
“Typical examples related to work on formative feedback have mostly focused on written feedback provided by generative AI [61] (e.g. aspects of writing supported by automated marking systems [40][41][42][43]).”
BackgroundResultInfluential
Metadata
- Authors
- Conijn, R., Kahr, P., & Snijders, C.
- Venue
- Journal of Learning Analytics
- Published
- 2023-03-12
- Volume
- 10
- Issue
- 1
- Pages
- 37-53
- ISSN
- 1929-7750 (ISSN-L)
- OpenAlex
- W4323670074
- Cited by
- 76
- References
- 36
- FWCI
- 6.68
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.